The aim of this project is to classify people's emotions based on their facial images. We have got more than 20000 images with their associated facial expressions labels and around 2000 images with their facial-keypoints annotations.

PROJECT OVERVIEW

alt text

We combined two models:

  • A facial key-point detection model for dtection of facial key-points
  • A Facial Expression(Emotion) detection model to detect the expresssion or emotion of individuals, facial images provided.
    The first part is deveoted to the facial key-points detection model.

PART 1: KEY FACIAL POINTS DETECTION

alt text

alt text

Importing packages and our dataset

In [1]:
#mounting the drive
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
In [3]:
#setting our working directory
%cd /content/drive/My Drive/Colab Notebooks/Modern AI Portfolio Builder/Emotion AI/Emotion_AI_Dataset/Emotion AI Dataset
/content/drive/My Drive/Colab Notebooks/Modern AI Portfolio Builder/Emotion AI/Emotion_AI_Dataset/Emotion AI Dataset
In [4]:
# importing libraries
import pandas as pd
import numpy as np
import os
import PIL
import seaborn as sns
import pickle
from PIL import *
import cv2
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.applications import DenseNet121
from tensorflow.keras.models import Model, load_model
from tensorflow.keras.initializers import glorot_uniform
from tensorflow.keras.utils import plot_model
from tensorflow.keras.callbacks import ReduceLROnPlateau, EarlyStopping, ModelCheckpoint, LearningRateScheduler
from IPython.display import display
from tensorflow.python.keras import *
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras import layers, optimizers
from tensorflow.keras.applications.resnet50 import ResNet50
from tensorflow.keras.layers import *
from tensorflow.keras import backend as K
from keras import optimizers
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from google.colab.patches import cv2_imshow
In [5]:
#loading my data
keyfacial_df=pd.read_csv('/content/drive/My Drive/Colab Notebooks/Modern AI Portfolio Builder/Emotion AI/Emotion_AI_Dataset/Emotion AI Dataset/data.csv')
In [6]:
keyfacial_df
Out[6]:
left_eye_center_x left_eye_center_y right_eye_center_x right_eye_center_y left_eye_inner_corner_x left_eye_inner_corner_y left_eye_outer_corner_x left_eye_outer_corner_y right_eye_inner_corner_x right_eye_inner_corner_y right_eye_outer_corner_x right_eye_outer_corner_y left_eyebrow_inner_end_x left_eyebrow_inner_end_y left_eyebrow_outer_end_x left_eyebrow_outer_end_y right_eyebrow_inner_end_x right_eyebrow_inner_end_y right_eyebrow_outer_end_x right_eyebrow_outer_end_y nose_tip_x nose_tip_y mouth_left_corner_x mouth_left_corner_y mouth_right_corner_x mouth_right_corner_y mouth_center_top_lip_x mouth_center_top_lip_y mouth_center_bottom_lip_x mouth_center_bottom_lip_y Image
0 66.033564 39.002274 30.227008 36.421678 59.582075 39.647423 73.130346 39.969997 36.356571 37.389402 23.452872 37.389402 56.953263 29.033648 80.227128 32.228138 40.227609 29.002322 16.356379 29.647471 44.420571 57.066803 61.195308 79.970165 28.614496 77.388992 43.312602 72.935459 43.130707 84.485774 238 236 237 238 240 240 239 241 241 243 240 23...
1 64.332936 34.970077 29.949277 33.448715 58.856170 35.274349 70.722723 36.187166 36.034723 34.361532 24.472511 33.144443 53.987404 28.275949 78.634213 30.405923 42.728851 26.146043 16.865362 27.058860 48.206298 55.660936 56.421447 76.352000 35.122383 76.047660 46.684596 70.266553 45.467915 85.480170 219 215 204 196 204 211 212 200 180 168 178 19...
2 65.057053 34.909642 30.903789 34.909642 59.412000 36.320968 70.984421 36.320968 37.678105 36.320968 24.976421 36.603221 55.742526 27.570947 78.887368 32.651621 42.193895 28.135453 16.791158 32.087116 47.557263 53.538947 60.822947 73.014316 33.726316 72.732000 47.274947 70.191789 47.274947 78.659368 144 142 159 180 188 188 184 180 167 132 84 59 ...
3 65.225739 37.261774 32.023096 37.261774 60.003339 39.127179 72.314713 38.380967 37.618643 38.754115 25.307270 38.007903 56.433809 30.929864 77.910261 31.665725 41.671513 31.049990 20.458017 29.909343 51.885078 54.166539 65.598887 72.703722 37.245496 74.195478 50.303165 70.091687 51.561183 78.268383 193 192 193 194 194 194 193 192 168 111 50 12 ...
4 66.725301 39.621261 32.244810 38.042032 58.565890 39.621261 72.515926 39.884466 36.982380 39.094852 22.506110 38.305237 57.249571 30.672177 77.762945 31.737247 38.035436 30.935382 15.925870 30.672177 43.299534 64.889521 60.671411 77.523239 31.191755 76.997301 44.962748 73.707387 44.227141 86.871166 147 148 160 196 215 214 216 217 219 220 206 18...
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2135 67.180378 35.816373 33.239956 34.921932 59.347973 37.000904 72.667896 37.097600 39.404349 36.589944 24.972418 36.348203 56.402841 31.171586 80.709708 30.595738 38.187845 30.595738 17.275454 29.534964 45.805390 60.065396 64.397610 73.248393 35.446431 74.014748 48.708626 76.760852 48.559612 77.335618 191 191 191 190 189 185 184 120 54 34 19 24 31...
2136 65.724490 36.301020 25.377551 37.311224 58.530612 37.739796 74.448980 37.525510 33.551020 38.107143 15.979592 38.076531 56.941270 29.605102 81.314059 30.754762 35.059410 29.566780 8.578912 32.096032 53.649433 62.472789 66.928121 79.229046 28.015377 81.151722 48.771976 81.816774 49.287271 82.871156 19 19 19 18 13 7 3 4 3 1 3 9 11 12 12 13 10 9 ...
2137 68.430866 38.651975 28.895857 37.617027 61.659350 40.100902 75.586792 40.219182 37.665118 39.087245 21.621652 39.095524 62.388142 32.089787 85.829396 38.724104 42.595762 30.541779 11.230184 34.485512 53.318891 63.289576 68.864397 77.495823 22.013981 77.368995 49.180628 79.043130 47.176739 89.544522 31 40 47 31 54 58 63 100 86 80 82 75 79 86 90 ...
2138 64.152180 30.691592 27.000898 40.868082 56.505624 34.126963 73.436776 28.556335 34.746122 40.506939 16.850204 44.409861 51.463469 27.161327 77.354898 19.562755 34.135510 32.830102 9.731633 41.152347 54.075935 59.735799 73.730743 70.273886 38.777143 80.684286 58.042857 79.301429 58.611086 80.355543 7 1 5 1 3 20 12 0 5 8 6 18 13 9 8 12 12 11 4 8...
2139 66.683755 34.483429 30.784490 38.578939 59.255347 36.065143 73.942694 34.624653 37.478531 39.398041 22.056816 40.330122 53.299787 29.987657 80.947787 27.977535 38.779298 33.302596 14.798890 35.982760 49.973878 59.269388 72.600433 71.862041 34.232759 77.339429 51.599453 75.963592 52.923371 82.661062 68 19 19 23 19 26 23 16 8 15 9 9 17 16 15 27 2...

2140 rows × 31 columns

In [7]:
#obtain relevant information about dataframe
keyfacial_df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2140 entries, 0 to 2139
Data columns (total 31 columns):
 #   Column                     Non-Null Count  Dtype  
---  ------                     --------------  -----  
 0   left_eye_center_x          2140 non-null   float64
 1   left_eye_center_y          2140 non-null   float64
 2   right_eye_center_x         2140 non-null   float64
 3   right_eye_center_y         2140 non-null   float64
 4   left_eye_inner_corner_x    2140 non-null   float64
 5   left_eye_inner_corner_y    2140 non-null   float64
 6   left_eye_outer_corner_x    2140 non-null   float64
 7   left_eye_outer_corner_y    2140 non-null   float64
 8   right_eye_inner_corner_x   2140 non-null   float64
 9   right_eye_inner_corner_y   2140 non-null   float64
 10  right_eye_outer_corner_x   2140 non-null   float64
 11  right_eye_outer_corner_y   2140 non-null   float64
 12  left_eyebrow_inner_end_x   2140 non-null   float64
 13  left_eyebrow_inner_end_y   2140 non-null   float64
 14  left_eyebrow_outer_end_x   2140 non-null   float64
 15  left_eyebrow_outer_end_y   2140 non-null   float64
 16  right_eyebrow_inner_end_x  2140 non-null   float64
 17  right_eyebrow_inner_end_y  2140 non-null   float64
 18  right_eyebrow_outer_end_x  2140 non-null   float64
 19  right_eyebrow_outer_end_y  2140 non-null   float64
 20  nose_tip_x                 2140 non-null   float64
 21  nose_tip_y                 2140 non-null   float64
 22  mouth_left_corner_x        2140 non-null   float64
 23  mouth_left_corner_y        2140 non-null   float64
 24  mouth_right_corner_x       2140 non-null   float64
 25  mouth_right_corner_y       2140 non-null   float64
 26  mouth_center_top_lip_x     2140 non-null   float64
 27  mouth_center_top_lip_y     2140 non-null   float64
 28  mouth_center_bottom_lip_x  2140 non-null   float64
 29  mouth_center_bottom_lip_y  2140 non-null   float64
 30  Image                      2140 non-null   object 
dtypes: float64(30), object(1)
memory usage: 518.4+ KB

The dataset contains one feature of type object this is pixel of face images

In [8]:
#check if null values exist in the dataframe
keyfacial_df.isnull().sum() # There is non null values in the dataset
Out[8]:
left_eye_center_x            0
left_eye_center_y            0
right_eye_center_x           0
right_eye_center_y           0
left_eye_inner_corner_x      0
left_eye_inner_corner_y      0
left_eye_outer_corner_x      0
left_eye_outer_corner_y      0
right_eye_inner_corner_x     0
right_eye_inner_corner_y     0
right_eye_outer_corner_x     0
right_eye_outer_corner_y     0
left_eyebrow_inner_end_x     0
left_eyebrow_inner_end_y     0
left_eyebrow_outer_end_x     0
left_eyebrow_outer_end_y     0
right_eyebrow_inner_end_x    0
right_eyebrow_inner_end_y    0
right_eyebrow_outer_end_x    0
right_eyebrow_outer_end_y    0
nose_tip_x                   0
nose_tip_y                   0
mouth_left_corner_x          0
mouth_left_corner_y          0
mouth_right_corner_x         0
mouth_right_corner_y         0
mouth_center_top_lip_x       0
mouth_center_top_lip_y       0
mouth_center_bottom_lip_x    0
mouth_center_bottom_lip_y    0
Image                        0
dtype: int64
In [9]:
keyfacial_df['Image'].shape # we have 2140 images with their facial keypoints annotations
Out[9]:
(2140,)

Since values for the images are given as space separated string, we separate the values using ' ' as separator. Then we convert them into numpy array np.formstring and convert the obtain 1D array into 2D array of shape(96,96).

In [10]:
keyfacial_df['Image']=keyfacial_df['Image'].apply(lambda x: np.fromstring(x, dtype=int, sep=' ').reshape(96,96))
In [11]:
#Obtain the shape of the Image
keyfacial_df['Image'][0].shape # The first image is 96X96
Out[11]:
(96, 96)
In [12]:
keyfacial_df.describe()
Out[12]:
left_eye_center_x left_eye_center_y right_eye_center_x right_eye_center_y left_eye_inner_corner_x left_eye_inner_corner_y left_eye_outer_corner_x left_eye_outer_corner_y right_eye_inner_corner_x right_eye_inner_corner_y right_eye_outer_corner_x right_eye_outer_corner_y left_eyebrow_inner_end_x left_eyebrow_inner_end_y left_eyebrow_outer_end_x left_eyebrow_outer_end_y right_eyebrow_inner_end_x right_eyebrow_inner_end_y right_eyebrow_outer_end_x right_eyebrow_outer_end_y nose_tip_x nose_tip_y mouth_left_corner_x mouth_left_corner_y mouth_right_corner_x mouth_right_corner_y mouth_center_top_lip_x mouth_center_top_lip_y mouth_center_bottom_lip_x mouth_center_bottom_lip_y
count 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000 2140.000000
mean 66.221549 36.842274 29.640269 37.063815 59.272128 37.856014 73.412473 37.640110 36.603107 37.920852 22.361617 38.034571 56.147991 29.222304 79.617523 29.656570 39.272084 29.413747 15.761707 30.452947 47.952141 57.253926 63.419076 75.887660 32.967365 76.134065 48.081325 72.681125 48.149654 82.630412
std 2.087683 2.294027 2.051575 2.234334 2.005631 2.034500 2.701639 2.684162 1.822784 2.009505 2.768804 2.654903 2.819914 2.867131 3.312647 3.627187 2.609648 2.842219 3.337901 3.644342 3.276053 4.528635 3.650131 4.438565 3.595103 4.259514 2.723274 5.108675 3.032389 4.813557
min 47.835757 23.832996 18.922611 24.773072 41.779381 27.190098 52.947144 26.250023 24.112624 26.250023 11.054589 26.521633 35.581733 15.859361 55.549929 10.522325 25.060327 16.476000 3.826243 13.224453 24.472590 41.558400 43.869480 57.023258 9.778137 56.690208 32.260312 56.719043 33.047605 57.232296
25% 65.046300 35.468842 28.472224 35.818377 58.113054 36.607950 71.741978 36.102409 35.495730 36.766783 20.631964 36.569765 54.562757 27.590944 77.732395 27.658656 37.555192 27.798773 13.540535 28.268263 46.495330 54.466000 61.341291 72.874263 30.879288 73.280038 46.580004 69.271669 46.492000 79.417480
50% 66.129065 36.913319 29.655440 37.048085 59.327154 37.845220 73.240045 37.624207 36.620735 37.920336 22.529600 37.876900 56.239984 29.468466 79.797607 29.753523 39.247681 29.528098 15.760876 30.333323 47.900511 57.638582 63.199057 75.682465 33.034022 75.941985 47.939031 72.395978 47.980854 82.388899
75% 67.332093 38.286438 30.858673 38.333884 60.521492 39.195431 74.978684 39.308331 37.665280 39.143921 24.202338 39.401034 57.926361 31.079433 81.559651 31.763791 40.824154 31.134740 17.871569 32.661300 49.260657 60.303524 65.302398 78.774969 35.063575 78.884031 49.290000 75.840286 49.551936 85.697976
max 78.013082 46.132421 42.495172 45.980981 69.023030 47.190316 87.032252 49.653825 47.293746 44.887301 40.050971 50.002113 67.752000 40.293408 94.269957 48.823425 51.300571 39.445859 39.676082 44.857962 65.279654 75.992731 84.767123 94.673637 50.973348 93.443176 61.804506 93.916338 62.438095 95.808983

Performing Image visualisation

Let's at first plot a random image from the dataset along with facial keypoints. Image data is obtain from df['Image'] and plotted using plt.imshow. x and y coordinates are obtain from the corresponding image. since x_coordinates are in even column like 0,2,4,... and y_coordinates are in odd column ones, we access their value using .loc command, which get the values for coordinates of the image based on the column it is refering.

In [13]:
i=np.random.randint(1,len(keyfacial_df))
plt.imshow(keyfacial_df['Image'][i], cmap='gray')
for j in range(1,31,2):
  plt.plot(keyfacial_df.loc[i][j-1], keyfacial_df.loc[i][j], 'rx')

Let's see the 16 first images along with their facial keypoints in a grid format.

In [14]:
fig= plt.figure(figsize=(20,20))

for i in range(16):
  ax=fig.add_subplot(4,4,i+1)
  image=plt.imshow(keyfacial_df['Image'][i], cmap='gray')
  for j in range(1,31,2):
    plt.plot(keyfacial_df.loc[i][j-1],keyfacial_df.loc[i][j], 'rx')
In [15]:
#This is for a rendom visualization
import random
fig= plt.figure(figsize=(20,20))

for i in range(64):
  k=random.randint(1,len(keyfacial_df)) #random image selection
  ax=fig.add_subplot(8,8,i+1)
  image=plt.imshow(keyfacial_df['Image'][k], cmap='gray')
  for j in range(1,31,2):
    plt.plot(keyfacial_df.loc[k][j-1],keyfacial_df.loc[k][j], 'rx')

Performing image augmentation

  • Here we're going to perform two type of augmentation:
    • Horizontal flipping: it flip the images along y -axis
    • Brightness augmenatation: we augment the brightness of images
      The images from the two types of augmentation are then added to our global dataset.
In [18]:
#Creating a new copy of the dataframe 
import copy
keyfacial_df_copy=copy.copy(keyfacial_df)
In [17]:
#Obtains the columns in the dataframe keyfacial_df
columns=keyfacial_df_copy.columns[:-1]
columns
Out[17]:
Index(['left_eye_center_x', 'left_eye_center_y', 'right_eye_center_x',
       'right_eye_center_y', 'left_eye_inner_corner_x',
       'left_eye_inner_corner_y', 'left_eye_outer_corner_x',
       'left_eye_outer_corner_y', 'right_eye_inner_corner_x',
       'right_eye_inner_corner_y', 'right_eye_outer_corner_x',
       'right_eye_outer_corner_y', 'left_eyebrow_inner_end_x',
       'left_eyebrow_inner_end_y', 'left_eyebrow_outer_end_x',
       'left_eyebrow_outer_end_y', 'right_eyebrow_inner_end_x',
       'right_eyebrow_inner_end_y', 'right_eyebrow_outer_end_x',
       'right_eyebrow_outer_end_y', 'nose_tip_x', 'nose_tip_y',
       'mouth_left_corner_x', 'mouth_left_corner_y', 'mouth_right_corner_x',
       'mouth_right_corner_y', 'mouth_center_top_lip_x',
       'mouth_center_top_lip_y', 'mouth_center_bottom_lip_x',
       'mouth_center_bottom_lip_y'],
      dtype='object')
  • Horizontal flipping
In [37]:
keyfacial_df_copy['Image']=keyfacial_df_copy['Image'].apply(lambda x: np.flip(x,axis=1))

Since we are flipping along y axis, y coordinates would be the same, only x coordinates would change. What we have to do then is to substract our initial x-coordinates values from width of the image.

In [38]:
for i in range(len(columns)):
  if i%2==0:
    keyfacial_df_copy[columns[i]]=keyfacial_df_copy[columns[i]].apply(lambda x: 96. - float(x))
In [33]:
fig.subplots?
In [39]:
#show the original image
fig=plt.figure()
axes=fig.subplots(nrows=1,ncols=2)

axes[0].imshow(keyfacial_df['Image'][0], cmap='gray')
for j in range(1,31,2):
  axes[0].plot(keyfacial_df.loc[0][j-1], keyfacial_df.loc[0][j], 'rx')
axes[0].set_title("Original Image")

axes[1].imshow(keyfacial_df_copy['Image'][0], cmap='gray')
for j in range(1,31,2):
  axes[1].plot(keyfacial_df_copy.loc[0][j-1], keyfacial_df_copy.loc[0][j], 'rx')
axes[1].set_title("Flipped image")
Out[39]:
Text(0.5, 1.0, 'Flipped image')
In [43]:
#concatenate the original dataframe with the flipped dataframe
augmented_df=np.concatenate((keyfacial_df,keyfacial_df_copy))
In [44]:
augmented_df.shape
Out[44]:
(4280, 31)
  • Randomingly increasing lightness of the images
    We multiply pixel values by random values between 1.5 and 2 to increase the brightness of the image, we clip the value between 0 and 255.
In [45]:
import random 
keyfacial_df_copy=copy.copy(keyfacial_df)
keyfacial_df_copy['Image']=keyfacial_df_copy['Image'].apply(lambda x: np.clip(random.uniform(1.5, 2)*x,0.0,255.0))
augmented_df=np.concatenate((augmented_df,keyfacial_df_copy))
augmented_df.shape
Out[45]:
(6420, 31)
In [46]:
#showing image with increased brightness
plt.imshow(keyfacial_df_copy['Image'][0], cmap='gray')
for j in range(1,31,2):
  plt.plot(keyfacial_df_copy.loc[0][j-1],keyfacial_df_copy.loc[0][j], 'rx')

Performing data normalization and training data preparation

  • Flattening the images
In [60]:
#obtain the images which is present in the 31 st column (since value start from 0, we refer to 31st column by 30)
img=augmented_df[:,30]
#normalize the images 
img=img/255.

#create an empty array of shape (x, 96,96,1) to feed the model
X=np.empty((len(img), 96, 96, 1))

#Iterate through the img list and add image value to the empty array after expanding it's dimension from (96, 96) to (96,96, 1)
for i in range(len(img)):
  X[i,]=np.expand_dims(img[i], axis=2)

#convert the array type to float32
X=np.asarray(X).astype(np.float32)
X.shape
Out[60]:
(6420, 96, 96, 1)
  • Obtaining the targets
In [61]:
#Obtain the value of x & y coordinates which are to be used as target
y=augmented_df[:,:30]
y=np.asarray(y).astype(np.float32)
y.shape
Out[61]:
(6420, 30)
  • Split the data into train and test set
In [62]:
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.1)
In [63]:
X_train.shape
Out[63]:
(5778, 96, 96, 1)

Defining our network architecture: ResNet architecture

alt text

alt text

alt text

In [66]:
def res_block(X, filter, stage):

  # Convolutional_block
  X_copy = X

  f1 , f2, f3 = filter

  # Main Path
  X = Conv2D(f1, (1,1),strides = (1,1), name ='res_'+str(stage)+'_conv_a', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = MaxPool2D((2,2))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_conv_a')(X)
  X = Activation('relu')(X) 

  X = Conv2D(f2, kernel_size = (3,3), strides =(1,1), padding = 'same', name ='res_'+str(stage)+'_conv_b', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_conv_b')(X)
  X = Activation('relu')(X) 

  X = Conv2D(f3, kernel_size = (1,1), strides =(1,1),name ='res_'+str(stage)+'_conv_c', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_conv_c')(X)


  # Short path
  X_copy = Conv2D(f3, kernel_size = (1,1), strides =(1,1),name ='res_'+str(stage)+'_conv_copy', kernel_initializer= glorot_uniform(seed = 0))(X_copy)
  X_copy = MaxPool2D((2,2))(X_copy)
  X_copy = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_conv_copy')(X_copy)

  # ADD
  X = Add()([X,X_copy])
  X = Activation('relu')(X)

  # Identity Block 1
  X_copy = X


  # Main Path
  X = Conv2D(f1, (1,1),strides = (1,1), name ='res_'+str(stage)+'_identity_1_a', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_identity_1_a')(X)
  X = Activation('relu')(X) 

  X = Conv2D(f2, kernel_size = (3,3), strides =(1,1), padding = 'same', name ='res_'+str(stage)+'_identity_1_b', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_identity_1_b')(X)
  X = Activation('relu')(X) 

  X = Conv2D(f3, kernel_size = (1,1), strides =(1,1),name ='res_'+str(stage)+'_identity_1_c', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_identity_1_c')(X)

  # ADD
  X = Add()([X,X_copy])
  X = Activation('relu')(X)

  # Identity Block 2
  X_copy = X


  # Main Path
  X = Conv2D(f1, (1,1),strides = (1,1), name ='res_'+str(stage)+'_identity_2_a', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_identity_2_a')(X)
  X = Activation('relu')(X) 

  X = Conv2D(f2, kernel_size = (3,3), strides =(1,1), padding = 'same', name ='res_'+str(stage)+'_identity_2_b', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_identity_2_b')(X)
  X = Activation('relu')(X) 

  X = Conv2D(f3, kernel_size = (1,1), strides =(1,1),name ='res_'+str(stage)+'_identity_2_c', kernel_initializer= glorot_uniform(seed = 0))(X)
  X = BatchNormalization(axis =3, name = 'bn_'+str(stage)+'_identity_2_c')(X)

  # ADD
  X = Add()([X,X_copy])
  X = Activation('relu')(X)

  return X
In [68]:
input_shape = (96, 96, 1)

# Input tensor shape
X_input = Input(input_shape)

# Zero-padding
X = ZeroPadding2D((3,3))(X_input)

# 1 - stage
X = Conv2D(64, (7,7), strides= (2,2), name = 'conv1', kernel_initializer= glorot_uniform(seed = 0))(X)
X = BatchNormalization(axis =3, name = 'bn_conv1')(X)
X = Activation('relu')(X)
X = MaxPooling2D((3,3), strides= (2,2))(X)

# 2 - stage
X = res_block(X, filter= [64,64,256], stage= 2)

# 3 - stage
X = res_block(X, filter= [128,128,512], stage= 3)


# Average Pooling
X = AveragePooling2D((2,2), name = 'Averagea_Pooling')(X)

# Final layer
X = Flatten()(X)
X = Dense(4096, activation = 'relu')(X)
X = Dropout(0.2)(X)
X = Dense(2048, activation = 'relu')(X)
X = Dropout(0.1)(X)
X = Dense(30, activation = 'relu')(X)


model_1_facialKeyPoints = Model( inputs= X_input, outputs = X)
model_1_facialKeyPoints.summary()
Model: "model_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_2 (InputLayer)            [(None, 96, 96, 1)]  0                                            
__________________________________________________________________________________________________
zero_padding2d_1 (ZeroPadding2D (None, 102, 102, 1)  0           input_2[0][0]                    
__________________________________________________________________________________________________
conv1 (Conv2D)                  (None, 48, 48, 64)   3200        zero_padding2d_1[0][0]           
__________________________________________________________________________________________________
bn_conv1 (BatchNormalization)   (None, 48, 48, 64)   256         conv1[0][0]                      
__________________________________________________________________________________________________
activation_19 (Activation)      (None, 48, 48, 64)   0           bn_conv1[0][0]                   
__________________________________________________________________________________________________
max_pooling2d_5 (MaxPooling2D)  (None, 23, 23, 64)   0           activation_19[0][0]              
__________________________________________________________________________________________________
res_2_conv_a (Conv2D)           (None, 23, 23, 64)   4160        max_pooling2d_5[0][0]            
__________________________________________________________________________________________________
max_pooling2d_6 (MaxPooling2D)  (None, 11, 11, 64)   0           res_2_conv_a[0][0]               
__________________________________________________________________________________________________
bn_2_conv_a (BatchNormalization (None, 11, 11, 64)   256         max_pooling2d_6[0][0]            
__________________________________________________________________________________________________
activation_20 (Activation)      (None, 11, 11, 64)   0           bn_2_conv_a[0][0]                
__________________________________________________________________________________________________
res_2_conv_b (Conv2D)           (None, 11, 11, 64)   36928       activation_20[0][0]              
__________________________________________________________________________________________________
bn_2_conv_b (BatchNormalization (None, 11, 11, 64)   256         res_2_conv_b[0][0]               
__________________________________________________________________________________________________
activation_21 (Activation)      (None, 11, 11, 64)   0           bn_2_conv_b[0][0]                
__________________________________________________________________________________________________
res_2_conv_copy (Conv2D)        (None, 23, 23, 256)  16640       max_pooling2d_5[0][0]            
__________________________________________________________________________________________________
res_2_conv_c (Conv2D)           (None, 11, 11, 256)  16640       activation_21[0][0]              
__________________________________________________________________________________________________
max_pooling2d_7 (MaxPooling2D)  (None, 11, 11, 256)  0           res_2_conv_copy[0][0]            
__________________________________________________________________________________________________
bn_2_conv_c (BatchNormalization (None, 11, 11, 256)  1024        res_2_conv_c[0][0]               
__________________________________________________________________________________________________
bn_2_conv_copy (BatchNormalizat (None, 11, 11, 256)  1024        max_pooling2d_7[0][0]            
__________________________________________________________________________________________________
add_6 (Add)                     (None, 11, 11, 256)  0           bn_2_conv_c[0][0]                
                                                                 bn_2_conv_copy[0][0]             
__________________________________________________________________________________________________
activation_22 (Activation)      (None, 11, 11, 256)  0           add_6[0][0]                      
__________________________________________________________________________________________________
res_2_identity_1_a (Conv2D)     (None, 11, 11, 64)   16448       activation_22[0][0]              
__________________________________________________________________________________________________
bn_2_identity_1_a (BatchNormali (None, 11, 11, 64)   256         res_2_identity_1_a[0][0]         
__________________________________________________________________________________________________
activation_23 (Activation)      (None, 11, 11, 64)   0           bn_2_identity_1_a[0][0]          
__________________________________________________________________________________________________
res_2_identity_1_b (Conv2D)     (None, 11, 11, 64)   36928       activation_23[0][0]              
__________________________________________________________________________________________________
bn_2_identity_1_b (BatchNormali (None, 11, 11, 64)   256         res_2_identity_1_b[0][0]         
__________________________________________________________________________________________________
activation_24 (Activation)      (None, 11, 11, 64)   0           bn_2_identity_1_b[0][0]          
__________________________________________________________________________________________________
res_2_identity_1_c (Conv2D)     (None, 11, 11, 256)  16640       activation_24[0][0]              
__________________________________________________________________________________________________
bn_2_identity_1_c (BatchNormali (None, 11, 11, 256)  1024        res_2_identity_1_c[0][0]         
__________________________________________________________________________________________________
add_7 (Add)                     (None, 11, 11, 256)  0           bn_2_identity_1_c[0][0]          
                                                                 activation_22[0][0]              
__________________________________________________________________________________________________
activation_25 (Activation)      (None, 11, 11, 256)  0           add_7[0][0]                      
__________________________________________________________________________________________________
res_2_identity_2_a (Conv2D)     (None, 11, 11, 64)   16448       activation_25[0][0]              
__________________________________________________________________________________________________
bn_2_identity_2_a (BatchNormali (None, 11, 11, 64)   256         res_2_identity_2_a[0][0]         
__________________________________________________________________________________________________
activation_26 (Activation)      (None, 11, 11, 64)   0           bn_2_identity_2_a[0][0]          
__________________________________________________________________________________________________
res_2_identity_2_b (Conv2D)     (None, 11, 11, 64)   36928       activation_26[0][0]              
__________________________________________________________________________________________________
bn_2_identity_2_b (BatchNormali (None, 11, 11, 64)   256         res_2_identity_2_b[0][0]         
__________________________________________________________________________________________________
activation_27 (Activation)      (None, 11, 11, 64)   0           bn_2_identity_2_b[0][0]          
__________________________________________________________________________________________________
res_2_identity_2_c (Conv2D)     (None, 11, 11, 256)  16640       activation_27[0][0]              
__________________________________________________________________________________________________
bn_2_identity_2_c (BatchNormali (None, 11, 11, 256)  1024        res_2_identity_2_c[0][0]         
__________________________________________________________________________________________________
add_8 (Add)                     (None, 11, 11, 256)  0           bn_2_identity_2_c[0][0]          
                                                                 activation_25[0][0]              
__________________________________________________________________________________________________
activation_28 (Activation)      (None, 11, 11, 256)  0           add_8[0][0]                      
__________________________________________________________________________________________________
res_3_conv_a (Conv2D)           (None, 11, 11, 128)  32896       activation_28[0][0]              
__________________________________________________________________________________________________
max_pooling2d_8 (MaxPooling2D)  (None, 5, 5, 128)    0           res_3_conv_a[0][0]               
__________________________________________________________________________________________________
bn_3_conv_a (BatchNormalization (None, 5, 5, 128)    512         max_pooling2d_8[0][0]            
__________________________________________________________________________________________________
activation_29 (Activation)      (None, 5, 5, 128)    0           bn_3_conv_a[0][0]                
__________________________________________________________________________________________________
res_3_conv_b (Conv2D)           (None, 5, 5, 128)    147584      activation_29[0][0]              
__________________________________________________________________________________________________
bn_3_conv_b (BatchNormalization (None, 5, 5, 128)    512         res_3_conv_b[0][0]               
__________________________________________________________________________________________________
activation_30 (Activation)      (None, 5, 5, 128)    0           bn_3_conv_b[0][0]                
__________________________________________________________________________________________________
res_3_conv_copy (Conv2D)        (None, 11, 11, 512)  131584      activation_28[0][0]              
__________________________________________________________________________________________________
res_3_conv_c (Conv2D)           (None, 5, 5, 512)    66048       activation_30[0][0]              
__________________________________________________________________________________________________
max_pooling2d_9 (MaxPooling2D)  (None, 5, 5, 512)    0           res_3_conv_copy[0][0]            
__________________________________________________________________________________________________
bn_3_conv_c (BatchNormalization (None, 5, 5, 512)    2048        res_3_conv_c[0][0]               
__________________________________________________________________________________________________
bn_3_conv_copy (BatchNormalizat (None, 5, 5, 512)    2048        max_pooling2d_9[0][0]            
__________________________________________________________________________________________________
add_9 (Add)                     (None, 5, 5, 512)    0           bn_3_conv_c[0][0]                
                                                                 bn_3_conv_copy[0][0]             
__________________________________________________________________________________________________
activation_31 (Activation)      (None, 5, 5, 512)    0           add_9[0][0]                      
__________________________________________________________________________________________________
res_3_identity_1_a (Conv2D)     (None, 5, 5, 128)    65664       activation_31[0][0]              
__________________________________________________________________________________________________
bn_3_identity_1_a (BatchNormali (None, 5, 5, 128)    512         res_3_identity_1_a[0][0]         
__________________________________________________________________________________________________
activation_32 (Activation)      (None, 5, 5, 128)    0           bn_3_identity_1_a[0][0]          
__________________________________________________________________________________________________
res_3_identity_1_b (Conv2D)     (None, 5, 5, 128)    147584      activation_32[0][0]              
__________________________________________________________________________________________________
bn_3_identity_1_b (BatchNormali (None, 5, 5, 128)    512         res_3_identity_1_b[0][0]         
__________________________________________________________________________________________________
activation_33 (Activation)      (None, 5, 5, 128)    0           bn_3_identity_1_b[0][0]          
__________________________________________________________________________________________________
res_3_identity_1_c (Conv2D)     (None, 5, 5, 512)    66048       activation_33[0][0]              
__________________________________________________________________________________________________
bn_3_identity_1_c (BatchNormali (None, 5, 5, 512)    2048        res_3_identity_1_c[0][0]         
__________________________________________________________________________________________________
add_10 (Add)                    (None, 5, 5, 512)    0           bn_3_identity_1_c[0][0]          
                                                                 activation_31[0][0]              
__________________________________________________________________________________________________
activation_34 (Activation)      (None, 5, 5, 512)    0           add_10[0][0]                     
__________________________________________________________________________________________________
res_3_identity_2_a (Conv2D)     (None, 5, 5, 128)    65664       activation_34[0][0]              
__________________________________________________________________________________________________
bn_3_identity_2_a (BatchNormali (None, 5, 5, 128)    512         res_3_identity_2_a[0][0]         
__________________________________________________________________________________________________
activation_35 (Activation)      (None, 5, 5, 128)    0           bn_3_identity_2_a[0][0]          
__________________________________________________________________________________________________
res_3_identity_2_b (Conv2D)     (None, 5, 5, 128)    147584      activation_35[0][0]              
__________________________________________________________________________________________________
bn_3_identity_2_b (BatchNormali (None, 5, 5, 128)    512         res_3_identity_2_b[0][0]         
__________________________________________________________________________________________________
activation_36 (Activation)      (None, 5, 5, 128)    0           bn_3_identity_2_b[0][0]          
__________________________________________________________________________________________________
res_3_identity_2_c (Conv2D)     (None, 5, 5, 512)    66048       activation_36[0][0]              
__________________________________________________________________________________________________
bn_3_identity_2_c (BatchNormali (None, 5, 5, 512)    2048        res_3_identity_2_c[0][0]         
__________________________________________________________________________________________________
add_11 (Add)                    (None, 5, 5, 512)    0           bn_3_identity_2_c[0][0]          
                                                                 activation_34[0][0]              
__________________________________________________________________________________________________
activation_37 (Activation)      (None, 5, 5, 512)    0           add_11[0][0]                     
__________________________________________________________________________________________________
Averagea_Pooling (AveragePoolin (None, 2, 2, 512)    0           activation_37[0][0]              
__________________________________________________________________________________________________
flatten_1 (Flatten)             (None, 2048)         0           Averagea_Pooling[0][0]           
__________________________________________________________________________________________________
dense_3 (Dense)                 (None, 4096)         8392704     flatten_1[0][0]                  
__________________________________________________________________________________________________
dropout_2 (Dropout)             (None, 4096)         0           dense_3[0][0]                    
__________________________________________________________________________________________________
dense_4 (Dense)                 (None, 2048)         8390656     dropout_2[0][0]                  
__________________________________________________________________________________________________
dropout_3 (Dropout)             (None, 2048)         0           dense_4[0][0]                    
__________________________________________________________________________________________________
dense_5 (Dense)                 (None, 30)           61470       dropout_3[0][0]                  
==================================================================================================
Total params: 18,016,286
Trainable params: 18,007,710
Non-trainable params: 8,576
__________________________________________________________________________________________________

Compiling and training our Deep Learning model

In [70]:
adam = tf.keras.optimizers.Adam(learning_rate = 0.0001, beta_1 = 0.9, beta_2 = 0.999, amsgrad = False)
model_1_facialKeyPoints.compile(loss = "mean_squared_error", optimizer = adam , metrics = ['accuracy'])
# Check this out for more information on Adam optimizer: https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam
In [71]:
# Defining a checkpointer: save the best model with least validation loss
checkpointer = ModelCheckpoint(filepath = "FacialKeyPoints_weights.hdf5", verbose = 1, save_best_only = True)

Now we train the model with just 2 epochs to show it works. Our Already trained model is going to be loaded after then.

In [72]:
#Training the model with just 2 epochs
history = model_1_facialKeyPoints.fit(X_train, y_train, batch_size = 32, epochs = 2, validation_split = 0.05, callbacks=[checkpointer])
Epoch 1/2
172/172 [==============================] - 13s 23ms/step - loss: 499.6272 - accuracy: 0.3584 - val_loss: 826.2732 - val_accuracy: 0.6990

Epoch 00001: val_loss improved from inf to 826.27319, saving model to FacialKeyPoints_weights.hdf5
Epoch 2/2
172/172 [==============================] - 3s 20ms/step - loss: 65.6068 - accuracy: 0.5634 - val_loss: 171.4797 - val_accuracy: 0.6990

Epoch 00002: val_loss improved from 826.27319 to 171.47972, saving model to FacialKeyPoints_weights.hdf5
  • We already have got 56% accuracy. We can save this model then.
In [73]:
# save the model architecture to json file for future use

model_json = model_1_facialKeyPoints.to_json()
with open("FacialKeyPoints-model.json","w") as json_file:
  json_file.write(model_json)

Assessing trained key facial points detection model performance

We are going to load our already trained model and with the learning architecture presented above.

In [74]:
# Loading the facial key point model already trained
with open('detection.json', 'r') as json_file:
    json_savedModel= json_file.read()
    
# load the model architecture 
model_1_facialKeyPoints = tf.keras.models.model_from_json(json_savedModel)
model_1_facialKeyPoints.load_weights('weights_keypoint.hdf5')
adam = tf.keras.optimizers.Adam(learning_rate=0.0001, beta_1=0.9, beta_2=0.999, amsgrad=False)
model_1_facialKeyPoints.compile(loss="mean_squared_error", optimizer= adam , metrics = ['accuracy'])

Evaluating the model with our test sets, we have got 83% accuracy, this is a great model !!

In [75]:
#Evaluate the model 
result=model_1_facialKeyPoints.evaluate(X_test,y_test)
print("Accuracy: {}".format(result[1]))
21/21 [==============================] - 1s 13ms/step - loss: 8.0475 - accuracy: 0.8182
Accuracy: 0.8302180767059326

PART 2: EXPRESSION DETECTION MODEL

alt text

alt text

Importing our data: Pixels

In [78]:
#Read the csv files for the facial expression data
facialexpression_df=pd.read_csv('icml_face_data.csv')
In [79]:
facialexpression_df.head()
Out[79]:
emotion pixels
0 0 70 80 82 72 58 58 60 63 54 58 60 48 89 115 121...
1 0 151 150 147 155 148 133 111 140 170 174 182 15...
2 2 24 32 36 30 32 23 19 20 30 41 21 22 32 34 21 1...
3 2 20 17 19 21 25 38 42 42 46 54 56 62 63 66 82 1...
4 3 77 78 79 79 78 75 60 55 47 48 58 73 77 79 57 5...
In [80]:
facialexpression_df[' pixels'][0] # the first image pixels, it is in a string format
Out[80]:
'70 80 82 72 58 58 60 63 54 58 60 48 89 115 121 119 115 110 98 91 84 84 90 99 110 126 143 153 158 171 169 172 169 165 129 110 113 107 95 79 66 62 56 57 61 52 43 41 65 61 58 57 56 69 75 70 65 56 54 105 146 154 151 151 155 155 150 147 147 148 152 158 164 172 177 182 186 189 188 190 188 180 167 116 95 103 97 77 72 62 55 58 54 56 52 44 50 43 54 64 63 71 68 64 52 66 119 156 161 164 163 164 167 168 170 174 175 176 178 179 183 187 190 195 197 198 197 198 195 191 190 145 86 100 90 65 57 60 54 51 41 49 56 47 38 44 63 55 46 52 54 55 83 138 157 158 165 168 172 171 173 176 179 179 180 182 185 187 189 189 192 197 200 199 196 198 200 198 197 177 91 87 96 58 58 59 51 42 37 41 47 45 37 35 36 30 41 47 59 94 141 159 161 161 164 170 171 172 176 178 179 182 183 183 187 189 192 192 194 195 200 200 199 199 200 201 197 193 111 71 108 69 55 61 51 42 43 56 54 44 24 29 31 45 61 72 100 136 150 159 163 162 163 170 172 171 174 177 177 180 187 186 187 189 192 192 194 195 196 197 199 200 201 200 197 201 137 58 98 92 57 62 53 47 41 40 51 43 24 35 52 63 75 104 129 143 149 158 162 164 166 171 173 172 174 178 178 179 187 188 188 191 193 194 195 198 199 199 197 198 197 197 197 201 164 52 78 87 69 58 56 50 54 39 44 42 26 31 49 65 91 119 134 145 147 152 159 163 167 171 170 169 174 178 178 179 187 187 185 187 190 188 187 191 197 201 199 199 200 197 196 197 182 58 62 77 61 60 55 49 59 52 54 44 22 30 47 68 102 123 136 144 148 150 153 157 167 172 173 170 171 177 179 178 186 190 186 189 196 193 191 194 190 190 192 197 201 203 199 194 189 69 48 74 56 60 57 50 59 59 51 41 20 34 47 79 111 132 139 143 145 147 150 151 160 169 172 171 167 171 177 177 174 180 182 181 192 196 189 192 198 195 194 196 198 201 202 195 189 70 39 69 61 61 61 53 59 59 45 40 26 40 61 93 124 135 138 142 144 146 151 152 158 165 168 168 165 161 164 173 172 167 172 167 180 198 198 193 199 195 194 198 200 198 197 195 190 65 35 68 59 59 62 57 60 59 50 44 32 54 90 115 132 137 138 140 144 146 146 156 165 168 174 176 176 175 168 168 169 171 175 171 172 192 194 184 198 205 201 194 195 193 195 192 186 57 38 72 65 57 62 58 57 60 54 49 47 79 116 130 138 141 141 139 141 143 145 157 164 164 166 173 174 176 179 179 176 181 189 188 173 180 175 160 182 189 198 192 189 190 190 188 172 46 44 64 66 59 62 57 56 62 53 50 66 103 133 137 141 143 141 136 132 131 136 127 118 111 107 108 123 131 143 154 158 166 177 181 175 170 159 148 171 161 176 185 192 194 188 190 162 53 49 58 63 61 61 55 56 61 51 50 81 116 139 142 142 146 144 136 128 119 112 97 85 90 91 88 92 90 80 81 84 106 122 132 144 145 144 147 163 147 163 173 181 190 187 191 167 61 48 53 61 61 58 54 56 61 51 53 89 123 140 144 145 146 147 136 122 107 99 95 92 90 87 83 76 67 52 46 52 63 69 83 96 119 132 148 159 136 137 143 138 143 152 156 156 70 48 50 59 61 57 54 54 61 52 56 93 124 135 140 144 148 150 140 125 114 101 80 54 56 54 41 41 33 40 39 35 49 60 63 74 107 129 147 147 116 111 100 77 76 86 108 111 73 49 50 60 62 60 57 55 63 59 56 89 121 134 139 146 151 152 150 141 127 111 96 77 85 70 32 31 37 91 65 50 48 59 73 83 112 136 155 130 60 46 38 40 43 81 116 91 72 52 48 58 62 62 59 53 61 59 52 85 114 134 140 147 154 159 158 153 145 143 150 126 121 125 68 45 89 137 95 70 78 75 95 109 131 153 171 94 23 16 32 82 82 65 113 77 71 54 48 56 62 62 60 53 60 56 52 75 108 133 141 149 158 166 169 167 163 156 155 146 112 119 134 127 142 140 121 117 129 114 120 129 146 174 191 98 46 33 33 109 147 98 109 67 73 55 50 56 64 64 61 58 61 53 54 64 106 129 140 148 159 169 175 176 174 165 159 156 145 120 115 124 127 131 133 141 147 142 141 147 161 182 202 154 114 96 100 158 158 153 123 61 76 57 48 56 64 64 63 62 61 54 55 44 97 131 137 147 158 168 177 181 183 179 170 168 169 165 155 152 151 152 154 162 165 158 153 158 168 187 206 186 147 135 144 145 152 178 115 57 74 58 48 58 64 63 63 59 63 55 53 66 104 130 132 144 153 162 170 180 185 187 181 178 182 180 177 173 171 171 177 176 172 164 161 167 164 185 207 197 173 152 141 141 161 191 104 54 69 60 48 57 65 62 60 57 64 55 50 94 111 124 130 135 150 159 163 172 179 184 184 178 178 177 173 171 174 177 178 176 169 165 161 163 161 180 205 201 183 171 177 178 180 194 101 55 65 60 47 55 65 63 59 58 63 57 52 90 105 117 122 130 143 153 157 163 171 174 182 183 182 178 174 175 175 177 175 172 163 161 159 157 162 178 200 201 188 181 172 177 187 198 98 57 63 61 48 52 61 64 63 60 65 57 51 95 104 113 117 127 136 145 152 156 162 162 165 173 177 182 183 183 180 181 177 165 153 154 152 153 160 174 193 200 188 185 180 182 192 196 101 60 60 56 49 50 60 66 64 62 64 59 53 99 104 111 112 118 132 142 147 155 158 160 159 162 171 176 184 186 183 180 169 154 141 135 145 155 164 180 196 205 188 189 188 189 193 192 98 61 64 55 49 49 60 66 63 64 63 60 57 99 105 108 112 113 125 139 143 150 155 158 164 169 174 176 182 183 182 177 163 141 133 147 151 164 170 185 200 210 194 188 192 186 185 180 88 64 67 60 46 50 59 65 64 64 64 59 56 101 103 108 109 109 118 134 143 143 147 155 159 166 171 174 177 179 178 172 153 129 143 161 159 166 171 186 197 207 203 185 191 183 179 164 73 67 67 66 48 50 57 65 65 63 64 61 57 103 108 114 112 110 115 128 138 144 145 152 156 159 164 168 172 172 169 161 139 125 147 156 161 162 164 180 188 188 197 185 187 181 180 137 65 70 68 70 52 47 53 62 65 63 65 61 58 105 109 112 120 113 112 122 134 141 149 150 153 155 159 164 167 167 162 152 134 115 126 119 106 99 109 141 158 150 155 175 184 176 175 106 63 70 68 68 50 46 50 57 63 63 64 61 59 107 110 110 117 117 114 117 128 137 147 148 150 153 156 161 162 163 156 150 148 105 70 45 26 25 47 73 74 79 128 177 180 173 157 77 66 68 67 68 52 49 51 56 62 62 62 62 60 101 107 108 114 115 114 117 125 134 143 148 149 152 154 158 160 158 155 160 158 132 88 73 73 64 52 66 91 138 160 174 173 171 125 64 67 63 64 68 54 50 49 54 60 60 60 62 60 98 105 105 109 111 114 117 125 131 139 145 148 153 153 156 157 156 161 168 165 153 139 122 115 105 89 103 150 182 161 171 173 162 89 64 64 62 64 69 56 48 49 56 58 60 59 62 60 89 99 108 106 109 111 119 120 125 134 140 146 152 153 153 153 156 159 162 160 150 136 129 133 133 122 133 148 178 168 168 175 132 61 67 66 65 63 69 57 47 50 55 58 59 61 62 60 89 96 105 107 105 107 117 120 123 124 133 141 149 153 151 145 151 145 139 140 138 128 126 124 129 125 136 142 164 172 168 168 87 58 67 63 62 61 69 57 39 44 55 56 59 63 62 62 84 91 92 98 102 103 113 119 121 118 128 138 146 151 147 142 140 128 127 128 129 126 135 140 135 130 143 146 149 166 174 131 62 65 62 59 67 63 68 83 89 65 42 52 60 60 62 63 77 84 84 91 99 101 107 112 117 118 122 134 145 149 144 134 127 127 129 130 134 125 126 132 152 153 151 150 151 165 171 87 59 65 64 61 58 86 122 138 208 207 154 71 52 56 55 56 69 77 83 85 93 91 102 112 116 118 119 127 140 144 142 131 112 95 85 75 62 58 56 59 87 88 83 127 142 165 149 62 65 62 59 77 113 192 156 84 185 196 197 168 81 70 75 69 58 65 73 82 81 79 95 107 114 116 116 123 136 142 136 132 131 102 71 58 49 41 33 41 36 49 60 99 136 168 111 53 63 71 138 186 203 195 146 87 91 72 79 95 103 82 61 74 55 57 68 75 76 77 84 96 106 110 111 121 130 138 136 142 153 159 152 152 154 145 133 136 147 158 156 155 147 158 74 57 60 123 181 174 126 89 72 67 57 43 55 67 76 86 60 45 51 45 52 68 75 73 77 88 96 100 104 113 115 121 134 146 149 146 149 148 155 168 174 179 178 169 169 174 161 131 44 47 82 150 168 136 104 75 66 80 67 58 48 54 68 88 121 102 51 45 38 53 66 65 70 86 92 96 102 103 109 116 130 136 136 133 136 138 137 135 128 130 143 158 165 164 147 87 62 74 123 160 170 100 99 107 79 71 86 75 57 45 49 65 122 130 43 48 40 39 55 61 59 71 82 87 88 93 105 118 123 128 130 124 111 98 94 88 67 55 84 129 147 148 105 48 82 142 161 164 164 76 72 85 100 88 72 90 84 54 48 54 73 100 73 36 44 31 37 53 51 55 67 74 77 87 97 108 118 125 132 122 106 86 80 82 75 73 83 110 129 126 46 22 130 177 196 193 166 72 52 54 73 100 92 75 99 95 65 68 61 63 91 65 42 37 22 28 39 44 57 68 74 83 92 101 119 131 143 141 134 136 140 139 134 136 139 138 136 85 23 114 202 198 199 180 173 98 36 86 130 150 137 99 77 101 99 72 56 43 77 82 79 70 56 28 20 25 36 50 63 73 83 98 111 124 139 156 160 159 169 168 165 163 159 149 114 43 26 133 183 192 177 152 137 130 125 139 173 195 186 137 101 88 101 105 70 46 77 72 84 87 87 81 64 37 20 31 40 46 65 88 108 110 125 149 157 153 162 164 158 159 154 140 78 21 11 61 144 168 173 157 138 150 148 132 159 182 183 136 106 116 95 106 109 82'
  • Function to convert pixel values in string format to array format
In [83]:
def string2array(x):
  return np.array(x.split(' ')).reshape(48,48,1).astype('float32')
  • Function to resize images from (48,48) to (96,96)
In [84]:
#Resizes images from (48? 48) to (96, 96)

def resize(x):
  img=x.reshape(48, 48)
  return cv2.resize(img, dsize=(96,96), interpolation=cv2.INTER_CUBIC)
  • Applying the above function to transform the data
In [85]:
facialexpression_df[' pixels']=facialexpression_df[' pixels'].apply(lambda x: string2array(x))
In [86]:
facialexpression_df[' pixels']=facialexpression_df[' pixels'].apply(lambda x: resize(x))
In [87]:
facialexpression_df.head()
Out[87]:
emotion pixels
0 0 [[69.316925, 73.03865, 79.13719, 84.17186, 85....
1 0 [[151.09435, 150.91393, 150.65791, 148.96367, ...
2 2 [[23.061905, 25.50914, 29.47847, 33.99843, 36....
3 2 [[20.083221, 19.079437, 17.398712, 17.158691, ...
4 3 [[76.26172, 76.54747, 77.001785, 77.7672, 78.4...
In [88]:
#Check the shape of the data
facialexpression_df.shape
Out[88]:
(24568, 2)
In [89]:
#checking the presence of null values in the data frame
facialexpression_df.isnull().sum()
Out[89]:
emotion    0
 pixels    0
dtype: int64

Visualization of images with their labels

  • Performing sanity check visualizing one image per emotion
In [90]:
label_to_text={0:'anger',1:'disgust',2:'sad',3:'happiness',4:'surprise'}
In [93]:
emotion = [0, 1, 2, 3, 4]

for i in emotion:
  data=facialexpression_df[facialexpression_df['emotion'] == i][:1] # choice of the first image from the i-labeled images
  img=data[' pixels'].item()
  img=img.reshape(96, 96)
  plt.figure()
  plt.title(label_to_text[i])
  plt.imshow(img,cmap='gray')

Let's check if our data is balanaced

In [102]:
facialexpression_df.emotion.value_counts().index
Out[102]:
Int64Index([3, 2, 0, 4, 1], dtype='int64')
In [103]:
facialexpression_df.emotion.value_counts()
Out[103]:
3    8989
2    6077
0    4953
4    4002
1     547
Name: emotion, dtype: int64
In [104]:
plt.figure(figsize=(10,10))
sns.barplot(x=facialexpression_df.emotion.value_counts().index,y=facialexpression_df.emotion.value_counts())
Out[104]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fd20779ccc0>

The disgust-labeled face images are few compared to other lables, we can consider upsampling images with this label.

Image preparation and data augmentation

Splitting the dataframe into features and labels

In [105]:
from keras.utils import to_categorical

X=facialexpression_df[' pixels']
y=to_categorical(facialexpression_df['emotion'])
In [106]:
X
Out[106]:
0        [[69.316925, 73.03865, 79.13719, 84.17186, 85....
1        [[151.09435, 150.91393, 150.65791, 148.96367, ...
2        [[23.061905, 25.50914, 29.47847, 33.99843, 36....
3        [[20.083221, 19.079437, 17.398712, 17.158691, ...
4        [[76.26172, 76.54747, 77.001785, 77.7672, 78.4...
                               ...                        
24563    [[-4.901001, 5.6496277, 22.477203, 49.366516, ...
24564    [[181.71645, 179.72063, 176.40785, 175.93037, ...
24565    [[178.1166, 176.83118, 174.74146, 172.61276, 1...
24566    [[16.883408, 16.953583, 17.081573, 16.53476, 1...
24567    [[30.199814, 29.574478, 28.545822, 27.846603, ...
Name:  pixels, Length: 24568, dtype: object
In [107]:
y
Out[107]:
array([[1., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0.],
       ...,
       [0., 0., 0., 1., 0.],
       [1., 0., 0., 0., 0.],
       [0., 0., 0., 1., 0.]], dtype=float32)
In [108]:
X=np.stack(X, axis=0)
In [109]:
X=X.reshape(24568,96,96,1)
In [110]:
print(X.shape,y.shape)
(24568, 96, 96, 1) (24568, 5)
  • Splitting the dataset into train, test and validation set
In [111]:
from sklearn.model_selection import train_test_split

X_train, X_Test, y_train,y_Test=train_test_split(X,y,test_size=0.1, shuffle=True)
X_val, X_Test,y_val, y_Test=train_test_split(X_Test,y_Test,test_size=0.5, shuffle=True)
In [112]:
print(X_val.shape,y_val.shape)
(1228, 96, 96, 1) (1228, 5)
  • Scaling the data
In [113]:
#Image pre_processing
X_train=X_train/255
X_val=X_val/255
X_Test=X_Test/255
In [114]:
print(X_Test.shape,y_Test.shape)
(1229, 96, 96, 1) (1229, 5)
In [ ]:
print(X_val.shape,y_val.shape)
(1228, 96, 96, 1) (1228, 5)
In [ ]:
print(X_train.shape,y_train.shape)
(22111, 96, 96, 1) (22111, 5)
  • Performing data augmentation with the function ImageDataGenerator
In [115]:
#Another way of performing data augmentation
train_datagen=ImageDataGenerator(
    rotation_range=15,
    width_shift_range=0.1,
    shear_range=0.1,
    zoom_range=0.1,
    horizontal_flip=True,
    vertical_flip=True,
    brightness_range=[1.1,1.5],
    fill_mode="nearest"
)

Building and training our Facial Expression Model

  • Architecture of the model
In [116]:
input_shape = (96, 96, 1)

# Input tensor shape
X_input = Input(input_shape)

# Zero-padding
X = ZeroPadding2D((3, 3))(X_input)

# 1 - stage
X = Conv2D(64, (7, 7), strides= (2, 2), name = 'conv1', kernel_initializer= glorot_uniform(seed = 0))(X)
X = BatchNormalization(axis =3, name = 'bn_conv1')(X)
X = Activation('relu')(X)
X = MaxPooling2D((3, 3), strides= (2, 2))(X)

# 2 - stage
X = res_block(X, filter= [64, 64, 256], stage= 2)

# 3 - stage
X = res_block(X, filter= [128, 128, 512], stage= 3)

# 4 - stage
# X = res_block(X, filter= [256, 256, 1024], stage= 4)

# Average Pooling
X = AveragePooling2D((4, 4), name = 'Averagea_Pooling')(X)

# Final layer
X = Flatten()(X)
X = Dense(5, activation = 'softmax', name = 'Dense_final', kernel_initializer= glorot_uniform(seed=0))(X)

model_2_emotion = Model( inputs= X_input, outputs = X, name = 'Resnet18')

model_2_emotion.summary()
Model: "Resnet18"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_3 (InputLayer)            [(None, 96, 96, 1)]  0                                            
__________________________________________________________________________________________________
zero_padding2d_2 (ZeroPadding2D (None, 102, 102, 1)  0           input_3[0][0]                    
__________________________________________________________________________________________________
conv1 (Conv2D)                  (None, 48, 48, 64)   3200        zero_padding2d_2[0][0]           
__________________________________________________________________________________________________
bn_conv1 (BatchNormalization)   (None, 48, 48, 64)   256         conv1[0][0]                      
__________________________________________________________________________________________________
activation_38 (Activation)      (None, 48, 48, 64)   0           bn_conv1[0][0]                   
__________________________________________________________________________________________________
max_pooling2d_10 (MaxPooling2D) (None, 23, 23, 64)   0           activation_38[0][0]              
__________________________________________________________________________________________________
res_2_conv_a (Conv2D)           (None, 23, 23, 64)   4160        max_pooling2d_10[0][0]           
__________________________________________________________________________________________________
max_pooling2d_11 (MaxPooling2D) (None, 11, 11, 64)   0           res_2_conv_a[0][0]               
__________________________________________________________________________________________________
bn_2_conv_a (BatchNormalization (None, 11, 11, 64)   256         max_pooling2d_11[0][0]           
__________________________________________________________________________________________________
activation_39 (Activation)      (None, 11, 11, 64)   0           bn_2_conv_a[0][0]                
__________________________________________________________________________________________________
res_2_conv_b (Conv2D)           (None, 11, 11, 64)   36928       activation_39[0][0]              
__________________________________________________________________________________________________
bn_2_conv_b (BatchNormalization (None, 11, 11, 64)   256         res_2_conv_b[0][0]               
__________________________________________________________________________________________________
activation_40 (Activation)      (None, 11, 11, 64)   0           bn_2_conv_b[0][0]                
__________________________________________________________________________________________________
res_2_conv_copy (Conv2D)        (None, 23, 23, 256)  16640       max_pooling2d_10[0][0]           
__________________________________________________________________________________________________
res_2_conv_c (Conv2D)           (None, 11, 11, 256)  16640       activation_40[0][0]              
__________________________________________________________________________________________________
max_pooling2d_12 (MaxPooling2D) (None, 11, 11, 256)  0           res_2_conv_copy[0][0]            
__________________________________________________________________________________________________
bn_2_conv_c (BatchNormalization (None, 11, 11, 256)  1024        res_2_conv_c[0][0]               
__________________________________________________________________________________________________
bn_2_conv_copy (BatchNormalizat (None, 11, 11, 256)  1024        max_pooling2d_12[0][0]           
__________________________________________________________________________________________________
add_12 (Add)                    (None, 11, 11, 256)  0           bn_2_conv_c[0][0]                
                                                                 bn_2_conv_copy[0][0]             
__________________________________________________________________________________________________
activation_41 (Activation)      (None, 11, 11, 256)  0           add_12[0][0]                     
__________________________________________________________________________________________________
res_2_identity_1_a (Conv2D)     (None, 11, 11, 64)   16448       activation_41[0][0]              
__________________________________________________________________________________________________
bn_2_identity_1_a (BatchNormali (None, 11, 11, 64)   256         res_2_identity_1_a[0][0]         
__________________________________________________________________________________________________
activation_42 (Activation)      (None, 11, 11, 64)   0           bn_2_identity_1_a[0][0]          
__________________________________________________________________________________________________
res_2_identity_1_b (Conv2D)     (None, 11, 11, 64)   36928       activation_42[0][0]              
__________________________________________________________________________________________________
bn_2_identity_1_b (BatchNormali (None, 11, 11, 64)   256         res_2_identity_1_b[0][0]         
__________________________________________________________________________________________________
activation_43 (Activation)      (None, 11, 11, 64)   0           bn_2_identity_1_b[0][0]          
__________________________________________________________________________________________________
res_2_identity_1_c (Conv2D)     (None, 11, 11, 256)  16640       activation_43[0][0]              
__________________________________________________________________________________________________
bn_2_identity_1_c (BatchNormali (None, 11, 11, 256)  1024        res_2_identity_1_c[0][0]         
__________________________________________________________________________________________________
add_13 (Add)                    (None, 11, 11, 256)  0           bn_2_identity_1_c[0][0]          
                                                                 activation_41[0][0]              
__________________________________________________________________________________________________
activation_44 (Activation)      (None, 11, 11, 256)  0           add_13[0][0]                     
__________________________________________________________________________________________________
res_2_identity_2_a (Conv2D)     (None, 11, 11, 64)   16448       activation_44[0][0]              
__________________________________________________________________________________________________
bn_2_identity_2_a (BatchNormali (None, 11, 11, 64)   256         res_2_identity_2_a[0][0]         
__________________________________________________________________________________________________
activation_45 (Activation)      (None, 11, 11, 64)   0           bn_2_identity_2_a[0][0]          
__________________________________________________________________________________________________
res_2_identity_2_b (Conv2D)     (None, 11, 11, 64)   36928       activation_45[0][0]              
__________________________________________________________________________________________________
bn_2_identity_2_b (BatchNormali (None, 11, 11, 64)   256         res_2_identity_2_b[0][0]         
__________________________________________________________________________________________________
activation_46 (Activation)      (None, 11, 11, 64)   0           bn_2_identity_2_b[0][0]          
__________________________________________________________________________________________________
res_2_identity_2_c (Conv2D)     (None, 11, 11, 256)  16640       activation_46[0][0]              
__________________________________________________________________________________________________
bn_2_identity_2_c (BatchNormali (None, 11, 11, 256)  1024        res_2_identity_2_c[0][0]         
__________________________________________________________________________________________________
add_14 (Add)                    (None, 11, 11, 256)  0           bn_2_identity_2_c[0][0]          
                                                                 activation_44[0][0]              
__________________________________________________________________________________________________
activation_47 (Activation)      (None, 11, 11, 256)  0           add_14[0][0]                     
__________________________________________________________________________________________________
res_3_conv_a (Conv2D)           (None, 11, 11, 128)  32896       activation_47[0][0]              
__________________________________________________________________________________________________
max_pooling2d_13 (MaxPooling2D) (None, 5, 5, 128)    0           res_3_conv_a[0][0]               
__________________________________________________________________________________________________
bn_3_conv_a (BatchNormalization (None, 5, 5, 128)    512         max_pooling2d_13[0][0]           
__________________________________________________________________________________________________
activation_48 (Activation)      (None, 5, 5, 128)    0           bn_3_conv_a[0][0]                
__________________________________________________________________________________________________
res_3_conv_b (Conv2D)           (None, 5, 5, 128)    147584      activation_48[0][0]              
__________________________________________________________________________________________________
bn_3_conv_b (BatchNormalization (None, 5, 5, 128)    512         res_3_conv_b[0][0]               
__________________________________________________________________________________________________
activation_49 (Activation)      (None, 5, 5, 128)    0           bn_3_conv_b[0][0]                
__________________________________________________________________________________________________
res_3_conv_copy (Conv2D)        (None, 11, 11, 512)  131584      activation_47[0][0]              
__________________________________________________________________________________________________
res_3_conv_c (Conv2D)           (None, 5, 5, 512)    66048       activation_49[0][0]              
__________________________________________________________________________________________________
max_pooling2d_14 (MaxPooling2D) (None, 5, 5, 512)    0           res_3_conv_copy[0][0]            
__________________________________________________________________________________________________
bn_3_conv_c (BatchNormalization (None, 5, 5, 512)    2048        res_3_conv_c[0][0]               
__________________________________________________________________________________________________
bn_3_conv_copy (BatchNormalizat (None, 5, 5, 512)    2048        max_pooling2d_14[0][0]           
__________________________________________________________________________________________________
add_15 (Add)                    (None, 5, 5, 512)    0           bn_3_conv_c[0][0]                
                                                                 bn_3_conv_copy[0][0]             
__________________________________________________________________________________________________
activation_50 (Activation)      (None, 5, 5, 512)    0           add_15[0][0]                     
__________________________________________________________________________________________________
res_3_identity_1_a (Conv2D)     (None, 5, 5, 128)    65664       activation_50[0][0]              
__________________________________________________________________________________________________
bn_3_identity_1_a (BatchNormali (None, 5, 5, 128)    512         res_3_identity_1_a[0][0]         
__________________________________________________________________________________________________
activation_51 (Activation)      (None, 5, 5, 128)    0           bn_3_identity_1_a[0][0]          
__________________________________________________________________________________________________
res_3_identity_1_b (Conv2D)     (None, 5, 5, 128)    147584      activation_51[0][0]              
__________________________________________________________________________________________________
bn_3_identity_1_b (BatchNormali (None, 5, 5, 128)    512         res_3_identity_1_b[0][0]         
__________________________________________________________________________________________________
activation_52 (Activation)      (None, 5, 5, 128)    0           bn_3_identity_1_b[0][0]          
__________________________________________________________________________________________________
res_3_identity_1_c (Conv2D)     (None, 5, 5, 512)    66048       activation_52[0][0]              
__________________________________________________________________________________________________
bn_3_identity_1_c (BatchNormali (None, 5, 5, 512)    2048        res_3_identity_1_c[0][0]         
__________________________________________________________________________________________________
add_16 (Add)                    (None, 5, 5, 512)    0           bn_3_identity_1_c[0][0]          
                                                                 activation_50[0][0]              
__________________________________________________________________________________________________
activation_53 (Activation)      (None, 5, 5, 512)    0           add_16[0][0]                     
__________________________________________________________________________________________________
res_3_identity_2_a (Conv2D)     (None, 5, 5, 128)    65664       activation_53[0][0]              
__________________________________________________________________________________________________
bn_3_identity_2_a (BatchNormali (None, 5, 5, 128)    512         res_3_identity_2_a[0][0]         
__________________________________________________________________________________________________
activation_54 (Activation)      (None, 5, 5, 128)    0           bn_3_identity_2_a[0][0]          
__________________________________________________________________________________________________
res_3_identity_2_b (Conv2D)     (None, 5, 5, 128)    147584      activation_54[0][0]              
__________________________________________________________________________________________________
bn_3_identity_2_b (BatchNormali (None, 5, 5, 128)    512         res_3_identity_2_b[0][0]         
__________________________________________________________________________________________________
activation_55 (Activation)      (None, 5, 5, 128)    0           bn_3_identity_2_b[0][0]          
__________________________________________________________________________________________________
res_3_identity_2_c (Conv2D)     (None, 5, 5, 512)    66048       activation_55[0][0]              
__________________________________________________________________________________________________
bn_3_identity_2_c (BatchNormali (None, 5, 5, 512)    2048        res_3_identity_2_c[0][0]         
__________________________________________________________________________________________________
add_17 (Add)                    (None, 5, 5, 512)    0           bn_3_identity_2_c[0][0]          
                                                                 activation_53[0][0]              
__________________________________________________________________________________________________
activation_56 (Activation)      (None, 5, 5, 512)    0           add_17[0][0]                     
__________________________________________________________________________________________________
Averagea_Pooling (AveragePoolin (None, 1, 1, 512)    0           activation_56[0][0]              
__________________________________________________________________________________________________
flatten_2 (Flatten)             (None, 512)          0           Averagea_Pooling[0][0]           
__________________________________________________________________________________________________
Dense_final (Dense)             (None, 5)            2565        flatten_2[0][0]                  
==================================================================================================
Total params: 1,174,021
Trainable params: 1,165,445
Non-trainable params: 8,576
__________________________________________________________________________________________________
  • Compiling and training the network
In [117]:
model_2_emotion.compile(optimizer="Adam", loss="categorical_crossentropy", metrics=["accuracy"])
  • Using early stopping to exit training if validation loss is not decreasing even after certain epochs(patience)
  • save the best model with lower validation loss
In [118]:
earlystopping=EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=60)
checkpointer=ModelCheckpoint(filepath="FacialExpression_weights.hdf5", verbose=1, save_best_only=True)

Training the model using just 5 epochs. The already trained model will be uploaded after then

In [119]:
history=model_2_emotion.fit(train_datagen.flow(X_train,y_train,batch_size=32), validation_data=(X_val,y_val), steps_per_epoch=len(X_train) // 64,
  epochs=5,callbacks=[checkpointer,earlystopping])
Epoch 1/5
345/345 [==============================] - 17s 43ms/step - loss: 1.6423 - accuracy: 0.3526 - val_loss: 1.7197 - val_accuracy: 0.2459

Epoch 00001: val_loss improved from inf to 1.71974, saving model to FacialExpression_weights.hdf5
Epoch 2/5
345/345 [==============================] - 14s 41ms/step - loss: 1.1896 - accuracy: 0.5104 - val_loss: 1.7568 - val_accuracy: 0.2459

Epoch 00002: val_loss did not improve from 1.71974
Epoch 3/5
345/345 [==============================] - 14s 41ms/step - loss: 1.0343 - accuracy: 0.5731 - val_loss: 1.7224 - val_accuracy: 0.2068

Epoch 00003: val_loss did not improve from 1.71974
Epoch 4/5
345/345 [==============================] - 14s 41ms/step - loss: 0.9536 - accuracy: 0.6178 - val_loss: 1.7057 - val_accuracy: 0.2459

Epoch 00004: val_loss improved from 1.71974 to 1.70570, saving model to FacialExpression_weights.hdf5
Epoch 5/5
345/345 [==============================] - 14s 41ms/step - loss: 0.9332 - accuracy: 0.6314 - val_loss: 1.9202 - val_accuracy: 0.2459

Epoch 00005: val_loss did not improve from 1.70570
In [120]:
#saving the model architecture to json file for future use

model_json=model_2_emotion.to_json()
with open("FacialExpression-model.json","w") as json_file:
  json_file.write(model_json)

Assessing the performance of the model

In [121]:
#Loading the emoton detection model already trained with many epochs:200 epochs
with open('emotion.json','r') as json_file:
  json_savedModel=json_file.read()

#load the model architecture
model_2_emotion=tf.keras.models.model_from_json(json_savedModel)
model_2_emotion.load_weights('weights_emotions.hdf5')
model_2_emotion.compile(optimizer="Adam", loss="categorical_crossentropy", metrics=["accuracy"])
  • The model's accuracy is 86% !!
In [122]:
#Evaluating the model
score=model_2_emotion.evaluate(X_Test,y_Test)
print('Test_Accuracy:{}'.format(score[1]))
39/39 [==============================] - 1s 10ms/step - loss: 0.3898 - accuracy: 0.8568
Test_Accuracy:0.8616761565208435
  • Determining the predicted classes
    • Probabilities of each class for each samples
In [123]:
predicted_classes=model_2_emotion.predict(X_Test)
predicted_classes # give the probabilities of each class for each sample
Out[123]:
array([[3.1000539e-04, 1.5766037e-06, 2.7798684e-03, 9.9404728e-01,
        2.8613028e-03],
       [3.3500865e-02, 5.2313579e-05, 9.6238679e-01, 3.3791878e-03,
        6.8075897e-04],
       [3.2251009e-03, 2.8244032e-05, 2.0568611e-03, 9.9186897e-01,
        2.8207886e-03],
       ...,
       [6.2231612e-01, 1.4579248e-04, 3.7671506e-01, 2.1718364e-05,
        8.0131058e-04],
       [3.7116501e-03, 3.5591944e-05, 6.9906287e-02, 9.1730845e-01,
        9.0380153e-03],
       [5.8803540e-02, 3.7911494e-04, 2.0242576e-03, 8.0720469e-02,
        8.5807264e-01]], dtype=float32)
  • output the class corresponding to the maximum probability
In [129]:
predicted_classes=np.argmax(model_2_emotion.predict(X_Test),axis=-1)
predicted_classes
Out[129]:
array([3, 2, 3, ..., 0, 3, 4])
In [130]:
y_true=np.argmax(y_Test,axis=-1)
y_true
Out[130]:
array([3, 2, 3, ..., 2, 3, 4])
  • Implementation of our confusion matrix
In [131]:
from sklearn.metrics import confusion_matrix
cm=confusion_matrix(y_true,predicted_classes)
plt.figure(figsize=(10,10))
sns.heatmap(cm, annot=True, cbar=False)
Out[131]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fd29cb6bc50>
  • classification report
    • Precision for all label are 80% upwards
    • Recall for all labels are upwards of 80% except for the label 1, recall that this label is associated to few samples compared to others.
In [ ]:
#Reporting the classification
from sklearn.metrics import classification_report
print(classification_report(y_true, predicted_classes))
              precision    recall  f1-score   support

           0       0.80      0.83      0.81       259
           1       0.82      0.61      0.70        23
           2       0.81      0.85      0.83       313
           3       0.92      0.95      0.93       442
           4       0.96      0.82      0.88       192

    accuracy                           0.87      1229
   macro avg       0.86      0.81      0.83      1229
weighted avg       0.87      0.87      0.87      1229

Let's visualize the dataset the predicted and true expressions with their associated image

In [132]:
L = 5
W = 5

fig, axes = plt.subplots(L, W, figsize = (24, 24))
axes = axes.ravel()

for i in np.arange(0, L*W):
    axes[i].imshow(X_Test[i].reshape(96,96), cmap = 'gray')
    axes[i].set_title('Prediction = {}\n True = {}'.format(label_to_text[predicted_classes[i]], label_to_text[y_true[i]]))
    axes[i].axis('off')

plt.subplots_adjust(wspace = 1)   

PART 3: COMBINATION OF EXPRESSION AND KEY POINTS DETECTION MODELS

  • A function to make prediction of the 2 models
In [133]:
def predict(X_Test):
  # Making prediction from the key point model
  df_predict=model_1_facialKeyPoints.predict(X_Test)
  # Makin prediction from the emotion model
  df_emotion=np.argmax(model_2_emotion.predict(X_Test), axis=-1)
  #Reshaping array from (856,) to (856,1)
  df_emotion=np.expand_dims(df_emotion, axis=1)

  #Converting the predictions into a dataframe
  df_predict=pd.DataFrame(df_predict, columns=columns)
  #Adding emotion into the predicted dataframe
  df_predict['emotion']=df_emotion

  return df_predict
  • Making prediction using the 2 models
In [134]:
df_predict=predict(X_test)
In [135]:
df_predict.head()
Out[135]:
left_eye_center_x left_eye_center_y right_eye_center_x right_eye_center_y left_eye_inner_corner_x left_eye_inner_corner_y left_eye_outer_corner_x left_eye_outer_corner_y right_eye_inner_corner_x right_eye_inner_corner_y right_eye_outer_corner_x right_eye_outer_corner_y left_eyebrow_inner_end_x left_eyebrow_inner_end_y left_eyebrow_outer_end_x left_eyebrow_outer_end_y right_eyebrow_inner_end_x right_eyebrow_inner_end_y right_eyebrow_outer_end_x right_eyebrow_outer_end_y nose_tip_x nose_tip_y mouth_left_corner_x mouth_left_corner_y mouth_right_corner_x mouth_right_corner_y mouth_center_top_lip_x mouth_center_top_lip_y mouth_center_bottom_lip_x mouth_center_bottom_lip_y emotion
0 67.240051 37.149746 30.336187 34.132305 61.063515 37.492916 72.800644 37.551949 35.494236 35.270386 24.050247 33.691540 59.414791 30.990301 79.507637 29.786333 39.941269 29.358971 17.387751 26.564030 46.681538 65.671555 59.197128 77.854012 29.382851 75.595238 44.417076 74.262703 43.685776 85.266754 0
1 66.289307 39.660915 32.264606 38.134438 58.384834 40.522266 73.891167 40.964493 38.400631 39.580399 25.306648 38.345070 53.494839 32.253098 79.832703 33.272015 37.959690 33.393509 21.016083 30.254042 39.189465 61.506516 62.486191 82.213531 32.744541 79.911293 44.002529 80.274887 44.332764 84.489151 0
2 63.934937 34.773899 30.599009 36.647530 58.080631 35.776108 69.117096 35.091614 36.133644 36.861305 24.749750 37.715683 55.165779 28.331396 74.531258 27.235703 37.078316 29.903015 18.227974 32.220997 48.196896 52.774937 61.530769 73.315292 37.506046 74.720642 48.821960 69.074196 49.480469 80.707916 0
3 67.810036 38.277092 29.588602 37.226593 61.177368 38.727547 74.101471 39.185997 35.719131 37.801334 22.903280 37.880592 57.021221 31.417866 80.218109 30.651669 38.524872 30.509256 16.624964 29.556803 48.451302 56.521481 62.875595 76.833176 33.249565 76.430611 47.841656 72.301643 47.767147 84.599289 3
4 67.072586 36.370773 25.297422 36.960873 59.477753 37.434319 74.690102 37.373356 32.806171 37.579800 17.308578 38.170559 55.090160 28.902948 82.132881 28.576193 36.112270 29.029194 9.180915 29.795927 46.823040 58.534546 64.038414 81.483902 31.463198 82.400276 47.519958 79.197403 47.715073 86.251450 2
In [136]:
df_predict.shape
Out[136]:
(642, 31)
  • A plot of a grid of 16 images along with their predicted emotion and facial key points
In [137]:
fig= plt.figure(figsize=(20,20))

for i in range(25):
  ax=fig.add_subplot(5,5,i+1)
  plt.imshow(X_test[i].squeeze(), cmap='gray')
  plt.title('predicted class'+ ':'+' '+ label_to_text[df_predict['emotion'][i]])
  for j in range(1,31,2):
    plt.plot(df_predict.loc[i][j-1], df_predict.loc[i][j], 'rx')
  • We now need to save our trained model and it has to be saved in a SavedModel format.
  • The model will have a version number and will be saved in a structured directory
  • tf.saved_model.save is a function used to build a saved model that is suitable for serving using Tensorflow Serving.
  • After the model is saved, we can now use TensorFlow Serving to start making inference requests using a specific version of our trained model "servable".
In [138]:
import json
import tensorflow.keras.backend as K

def deploy(directory, model):
  MODEL_DIR=directory
  version=1

  #let's join the tmp model directory with our chosen version number
  #The expected result will be = '\tmp\version_number'
  export_path=os.path.join(MODEL_DIR, str(version))
  print('export_path={}\n'.format(export_path))

  # Let's save the model using saved_model.save
  # If the directory already exist, we will remove it using '!rm'
  # rm removes each file specified in the command line.

  if os.path.isdir(export_path):
      print('\nAlready saved a model, cleaning up\n')
      !rm -r {export_path}

  tf.saved_model.save(model, export_path)

  os.environ["MODEL_DIR"]=MODEL_DIR

Serving the model using Tensoflow serving

In [139]:
# Let's add tensorflow-model-server package to our list of packages 
!echo "deb http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | tee /etc/apt/sources.list.d/tensorflow-serving.list && \
curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | apt-key add -
!apt update
deb http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2943  100  2943    0     0  31645      0 --:--:-- --:--:-- --:--:-- 31645
OK
Get:1 http://storage.googleapis.com/tensorflow-serving-apt stable InRelease [3,012 B]
Ign:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease
Ign:3 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease
Get:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release [697 B]
Get:5 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release [564 B]
Get:6 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release.gpg [836 B]
Get:7 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release.gpg [833 B]
Get:8 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease [3,626 B]
Get:9 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:10 http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server-universal amd64 Packages [346 B]
Get:11 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease [15.9 kB]
Hit:12 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:13 http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server amd64 Packages [340 B]
Ign:14 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages
Get:14 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages [506 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:16 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Packages [66.1 kB]
Get:17 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ Packages [40.7 kB]
Get:18 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease [21.3 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:20 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main Sources [1,700 kB]
Get:21 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [237 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2,244 kB]
Get:23 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [15.3 kB]
Get:24 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,816 kB]
Get:25 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,372 kB]
Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [266 kB]
Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2,136 kB]
Get:28 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic/main amd64 Packages [870 kB]
Get:29 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [53.8 kB]
Get:30 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic/main amd64 Packages [46.5 kB]
Fetched 11.7 MB in 4s (3,116 kB/s)
Reading package lists... Done
Building dependency tree       
Reading state information... Done
59 packages can be upgraded. Run 'apt list --upgradable' to see them.
In [140]:
# Let's install tensorflow model server
!apt-get install tensorflow-model-server
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  tensorflow-model-server
0 upgraded, 1 newly installed, 0 to remove and 59 not upgraded.
Need to get 223 MB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://storage.googleapis.com/tensorflow-serving-apt stable/tensorflow-model-server amd64 tensorflow-model-server all 2.4.0 [223 MB]
Fetched 223 MB in 3s (82.6 MB/s)
Selecting previously unselected package tensorflow-model-server.
(Reading database ... 144865 files and directories currently installed.)
Preparing to unpack .../tensorflow-model-server_2.4.0_all.deb ...
Unpacking tensorflow-model-server (2.4.0) ...
Setting up tensorflow-model-server (2.4.0) ...

Let's run TensorFlow serving

Deployment of the model

In [141]:
deploy('/model', model_1_facialKeyPoints)
export_path=/model/1

INFO:tensorflow:Assets written to: /model/1/assets
In [142]:
%%bash --bg 
nohup tensorflow_model_server \
  --rest_api_port=4500 \
  --model_name=keypoint_model \
  --model_base_path="${MODEL_DIR}" >server.log 2>&1
Starting job # 0 in a separate thread.
In [143]:
!tail server.log
2020-12-21 20:09:16.725740: W external/org_tensorflow/tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 33554432 exceeds 10% of free system memory.
2020-12-21 20:09:16.745577: W external/org_tensorflow/tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 33554432 exceeds 10% of free system memory.
2020-12-21 20:09:16.769145: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: /model/1
2020-12-21 20:09:16.807800: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 283736 microseconds.
2020-12-21 20:09:16.815791: I tensorflow_serving/servables/tensorflow/saved_model_warmup_util.cc:59] No warmup data file found at /model/1/assets.extra/tf_serving_warmup_requests
2020-12-21 20:09:16.816047: I tensorflow_serving/core/loader_harness.cc:87] Successfully loaded servable version {name: keypoint_model version: 1}
2020-12-21 20:09:16.817906: I tensorflow_serving/model_servers/server.cc:371] Running gRPC ModelServer at 0.0.0.0:8500 ...
[warn] getaddrinfo: address family for nodename not supported
[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...
2020-12-21 20:09:16.818651: I tensorflow_serving/model_servers/server.cc:391] Exporting HTTP/REST API at:localhost:4500 ...
In [ ]:
deploy('/model1', model_2_emotion)
export_path=/model1/1

INFO:tensorflow:Assets written to: /model1/1/assets
In [ ]:
%%bash --bg 
nohup tensorflow_model_server \
  --rest_api_port=4000 \
  --model_name=emotion_model \
  --model_base_path="${MODEL_DIR}" >server.log 2>&1
Starting job # 17 in a separate thread.
In [ ]:
!tail server.log 
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-11-27 23:04:36.047917: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:199] Restoring SavedModel bundle.
2020-11-27 23:04:36.159797: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:183] Running initialization op on SavedModel bundle at path: /model1/1
2020-11-27 23:04:36.203204: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:303] SavedModel load for tags { serve }; Status: success: OK. Took 216307 microseconds.
2020-11-27 23:04:36.210745: I tensorflow_serving/servables/tensorflow/saved_model_warmup_util.cc:59] No warmup data file found at /model1/1/assets.extra/tf_serving_warmup_requests
2020-11-27 23:04:36.211061: I tensorflow_serving/core/loader_harness.cc:87] Successfully loaded servable version {name: emotion_model version: 1}
2020-11-27 23:04:36.212915: I tensorflow_serving/model_servers/server.cc:367] Running gRPC ModelServer at 0.0.0.0:8500 ...
[warn] getaddrinfo: address family for nodename not supported
[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...
2020-11-27 23:04:36.213859: I tensorflow_serving/model_servers/server.cc:387] Exporting HTTP/REST API at:localhost:4000 ...
  • Now we have successfully loaded a servable version of our model {name: keypoint_model version: 1}

Let's then make an inference

In [ ]:
import json

#Let's create a json object and make free inference requests

data=json.dumps({"signature_name": "serving_default","instances":X_test[0:3].tolist()})
print('Data: {} ... {}'.format(data[:50], data[len(data)-52:]))
Data: {"signature_name": "serving_default", "instances": ... 35], [0.45490196347236633], [0.5529412031173706]]]]}
In [ ]:
import requests

# Function to make predictions from deployed models
def response(data):
  headers = {"content-type": "application/json"}
  json_response = requests.post('http://localhost:4500/v1/models/keypoint_model/versions/1:predict', data=data, headers=headers, verify = False)
  df_predict = json.loads(json_response.text)['predictions']
  json_response = requests.post('http://localhost:4000/v1/models/emotion_model/versions/1:predict', data=data, headers=headers, verify = False)
  df_emotion = np.argmax(json.loads(json_response.text)['predictions'], axis = 1)
  
  # Reshaping array from (856,) to (856,1)
  df_emotion = np.expand_dims(df_emotion, axis = 1)

  # Converting the predictions into a dataframe
  df_predict= pd.DataFrame(df_predict, columns = columns)

  # Adding emotion into the predicted dataframe
  df_predict['emotion'] = df_emotion

  return df_predict
In [ ]:
# making prediction
df_predict = response(data)
In [ ]:
df_predict
Out[ ]:
left_eye_center_x left_eye_center_y right_eye_center_x right_eye_center_y left_eye_inner_corner_x left_eye_inner_corner_y left_eye_outer_corner_x left_eye_outer_corner_y right_eye_inner_corner_x right_eye_inner_corner_y right_eye_outer_corner_x right_eye_outer_corner_y left_eyebrow_inner_end_x left_eyebrow_inner_end_y left_eyebrow_outer_end_x left_eyebrow_outer_end_y right_eyebrow_inner_end_x right_eyebrow_inner_end_y right_eyebrow_outer_end_x right_eyebrow_outer_end_y nose_tip_x nose_tip_y mouth_left_corner_x mouth_left_corner_y mouth_right_corner_x mouth_right_corner_y mouth_center_top_lip_x mouth_center_top_lip_y mouth_center_bottom_lip_x mouth_center_bottom_lip_y emotion
0 67.422890 36.146145 29.792599 34.825119 60.219238 36.765904 74.467621 37.441067 36.927822 35.784668 21.944384 35.631863 57.064056 26.489881 81.474587 29.672941 41.573490 25.634544 14.908765 28.215477 49.007420 55.952427 62.443741 76.541168 31.390108 75.634872 47.408344 71.458763 47.155266 82.050652 2
1 67.864388 37.948967 28.598495 37.282524 60.803787 38.404877 74.565178 39.199837 35.177181 37.656727 21.544495 38.312504 58.145760 28.192144 81.297523 30.028284 35.676582 27.973822 14.798652 29.865740 47.065903 53.520474 62.425316 79.413429 34.424488 79.304489 47.388393 71.911758 47.878100 86.335600 4
2 65.509369 40.126438 30.307700 35.934479 58.527920 40.079174 71.950249 41.770157 36.370933 37.296833 23.197670 35.973949 55.926647 31.405666 79.842000 35.435562 40.374603 29.845516 16.751827 28.424978 44.028164 57.660168 58.263580 75.978462 27.990610 72.449486 42.759468 72.122620 42.104858 78.457146 0
In [ ]:
# Plotting the test images and their predicted keypoints and emotions

fig, axes = plt.subplots(3, 1, figsize = (24, 24))
axes = axes.ravel()

for i in range(3):
    axes[i].imshow(X_test[i].squeeze(), cmap='gray')
    axes[i].set_title('Prediction = {}'.format(label_to_text[df_predict['emotion'][i]]))
    axes[i].axis('off')

    for j in range(1,31,2):
            axes[i].plot(df_predict.loc[i][j-1], df_predict.loc[i][j], 'rx')
            

We have successfully deployed and made inference with our model running tensorflow serving.
This project is a beginnning of a list computer vision projects coming very soon so, stay tune if you want to know more about it.